{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# XGBoost Parameter Tuning for RentalListingInquiries Dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 第2步：调整正则化参数：reg_alpha 和 reg_lambda\n",
    "\n",
    "\n",
    "首先import必要的模块"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "from xgboost import XGBClassifier\n",
    "import xgboost as xgb\n",
    "\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "\n",
    "from sklearn.model_selection import GridSearchCV\n",
    "from sklearn.model_selection import StratifiedKFold\n",
    "\n",
    "from sklearn.metrics import log_loss\n",
    "\n",
    "from matplotlib import pyplot\n",
    "import seaborn as sns\n",
    "%matplotlib inline\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 读取数据"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# path to wehre the data lies\n",
    "dpath = \"D:/AI/week3data_RentalListingInquiries/\"\n",
    "data_train = pd.read_csv(dpath+\"RentListingInquries_FE_train.csv\")\n",
    "\n",
    "data_test = pd.read_csv(dpath+\"RentListingInquries_FE_test.csv\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Interest Level分布，看看各类样本分布是否均衡"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "#sns.countplot(data_train.interest_level)\n",
    "#pyplot.xlabel('interest_level')\n",
    "#pyplot.ylabel('Number of occurences')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "每类样本分布不是很均匀，所以交叉验证时可以考虑各类样本按比例抽取\n",
    "\n",
    "# 准备数据"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "y_train = data_train['interest_level']\n",
    "X_train = data_train.drop('interest_level',axis = 1)\n",
    "X_test = data_test"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "各类样本不均衡，交叉验证是采用StratifiedKFlod，在每次采样是各类样本按比例采样"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "# prepare cross validation\n",
    "kfold = StratifiedKFold(n_splits=5,shuffle=True,random_state=3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "第一轮参数调整得到的n_estimators最优值为220\n",
    "\n",
    "用交叉验证评价模型性能时，用scoring参数定义评价指标。评价指标是越高越好，因此用一些损失函数当评价指标时，需要再加负号，如neg_log_loss，neg_mean_squared_error详见sklearn文档"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'reg_alpha': [1.5, 2], 'reg_lambda': [0.5, 1, 2]}"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "reg_alpha = [1.5,2]\n",
    "reg_lambda = [0.5,1,2]\n",
    "\n",
    "param_test4_1 = dict(reg_alpha=reg_alpha, reg_lambda=reg_lambda)\n",
    "param_test4_1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "D:\\ProgramFiles\\Anaconda2\\lib\\site-packages\\sklearn\\model_selection\\_search.py:761: DeprecationWarning: The grid_scores_ attribute was deprecated in version 0.18 in favor of the more elaborate cv_results_ attribute. The grid_scores_ attribute will not be available from 0.20\n",
      "  DeprecationWarning)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "([mean: -0.58915, std: 0.00395, params: {'reg_alpha': 1.5, 'reg_lambda': 0.5},\n",
       "  mean: -0.58864, std: 0.00370, params: {'reg_alpha': 1.5, 'reg_lambda': 1},\n",
       "  mean: -0.58938, std: 0.00356, params: {'reg_alpha': 1.5, 'reg_lambda': 2},\n",
       "  mean: -0.58881, std: 0.00372, params: {'reg_alpha': 2, 'reg_lambda': 0.5},\n",
       "  mean: -0.58920, std: 0.00353, params: {'reg_alpha': 2, 'reg_lambda': 1},\n",
       "  mean: -0.58925, std: 0.00335, params: {'reg_alpha': 2, 'reg_lambda': 2}],\n",
       " {'reg_alpha': 1.5, 'reg_lambda': 1},\n",
       " -0.5886399898097323)"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "xgb4_1 = XGBClassifier(\n",
    "    learning_rate=0.1,\n",
    "    n_estimators=220, # 第一轮参数调整得到的n_estimators最优值\n",
    "    max_depth=5,\n",
    "    min_child_weight=1,\n",
    "    gamma=0,\n",
    "    subsample=0.3,\n",
    "    colsample_bytree=0.8,\n",
    "    colsample_bylevel=0.7,\n",
    "    objective='multi:softprob',\n",
    "    seed=3)\n",
    "\n",
    "gsearch4_1 = GridSearchCV(xgb4_1,param_grid=param_test4_1,scoring='neg_log_loss',n_jobs=-1,cv=kfold)\n",
    "gsearch4_1.fit(X_train,y_train)\n",
    "\n",
    "gsearch4_1.grid_scores_, gsearch4_1.best_params_, gsearch4_1.best_score_\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "D:\\ProgramFiles\\Anaconda2\\lib\\site-packages\\sklearn\\utils\\deprecation.py:122: FutureWarning: You are accessing a training score ('mean_train_score'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n",
      "D:\\ProgramFiles\\Anaconda2\\lib\\site-packages\\sklearn\\utils\\deprecation.py:122: FutureWarning: You are accessing a training score ('split0_train_score'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n",
      "D:\\ProgramFiles\\Anaconda2\\lib\\site-packages\\sklearn\\utils\\deprecation.py:122: FutureWarning: You are accessing a training score ('split1_train_score'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n",
      "D:\\ProgramFiles\\Anaconda2\\lib\\site-packages\\sklearn\\utils\\deprecation.py:122: FutureWarning: You are accessing a training score ('split2_train_score'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n",
      "D:\\ProgramFiles\\Anaconda2\\lib\\site-packages\\sklearn\\utils\\deprecation.py:122: FutureWarning: You are accessing a training score ('split3_train_score'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n",
      "D:\\ProgramFiles\\Anaconda2\\lib\\site-packages\\sklearn\\utils\\deprecation.py:122: FutureWarning: You are accessing a training score ('split4_train_score'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n",
      "D:\\ProgramFiles\\Anaconda2\\lib\\site-packages\\sklearn\\utils\\deprecation.py:122: FutureWarning: You are accessing a training score ('std_train_score'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "{'mean_fit_time': array([133.06739998, 131.22479997, 131.15840001, 131.39300003,\n",
       "        133.27220001, 131.72120004]),\n",
       " 'mean_score_time': array([0.55539994, 0.65560002, 0.54899998, 0.53720002, 0.56459994,\n",
       "        0.56459999]),\n",
       " 'mean_test_score': array([-0.58914533, -0.58863999, -0.58937752, -0.5888114 , -0.58919512,\n",
       "        -0.58925376]),\n",
       " 'mean_train_score': array([-0.5165805 , -0.51831375, -0.52085786, -0.52000675, -0.5213971 ,\n",
       "        -0.52355846]),\n",
       " 'param_reg_alpha': masked_array(data=[1.5, 1.5, 1.5, 2, 2, 2],\n",
       "              mask=[False, False, False, False, False, False],\n",
       "        fill_value='?',\n",
       "             dtype=object),\n",
       " 'param_reg_lambda': masked_array(data=[0.5, 1, 2, 0.5, 1, 2],\n",
       "              mask=[False, False, False, False, False, False],\n",
       "        fill_value='?',\n",
       "             dtype=object),\n",
       " 'params': [{'reg_alpha': 1.5, 'reg_lambda': 0.5},\n",
       "  {'reg_alpha': 1.5, 'reg_lambda': 1},\n",
       "  {'reg_alpha': 1.5, 'reg_lambda': 2},\n",
       "  {'reg_alpha': 2, 'reg_lambda': 0.5},\n",
       "  {'reg_alpha': 2, 'reg_lambda': 1},\n",
       "  {'reg_alpha': 2, 'reg_lambda': 2}],\n",
       " 'rank_test_score': array([3, 1, 6, 2, 4, 5]),\n",
       " 'split0_test_score': array([-0.58212293, -0.58241262, -0.58364696, -0.58268586, -0.58298218,\n",
       "        -0.58366376]),\n",
       " 'split0_train_score': array([-0.51808024, -0.52009783, -0.52294139, -0.52278408, -0.5230295 ,\n",
       "        -0.52544603]),\n",
       " 'split1_test_score': array([-0.58773057, -0.58704075, -0.58746817, -0.58691217, -0.58840725,\n",
       "        -0.58754921]),\n",
       " 'split1_train_score': array([-0.51693942, -0.51877524, -0.5210567 , -0.51946373, -0.52117783,\n",
       "        -0.52313347]),\n",
       " 'split2_test_score': array([-0.59059075, -0.58912327, -0.58978316, -0.5897551 , -0.58944892,\n",
       "        -0.59027775]),\n",
       " 'split2_train_score': array([-0.51617428, -0.51811925, -0.52036973, -0.5194083 , -0.52155083,\n",
       "        -0.52317333]),\n",
       " 'split3_test_score': array([-0.5928447 , -0.59223708, -0.59244936, -0.59131426, -0.59225719,\n",
       "        -0.59155354]),\n",
       " 'split3_train_score': array([-0.51687785, -0.51818762, -0.52051062, -0.51959738, -0.52093989,\n",
       "        -0.5242079 ]),\n",
       " 'split4_test_score': array([-0.59243869, -0.59238737, -0.59354122, -0.59339102, -0.59288121,\n",
       "        -0.59322576]),\n",
       " 'split4_train_score': array([-0.51483071, -0.51638881, -0.51941085, -0.51878028, -0.52028742,\n",
       "        -0.52183157]),\n",
       " 'std_fit_time': array([2.24092911, 0.77561857, 2.51249068, 1.65839165, 3.22070967,\n",
       "        3.16210748]),\n",
       " 'std_score_time': array([0.05892576, 0.15017938, 0.00599995, 0.00733209, 0.03588647,\n",
       "        0.07211266]),\n",
       " 'std_test_score': array([0.00394811, 0.00370279, 0.00356148, 0.00372169, 0.00352843,\n",
       "        0.00335459]),\n",
       " 'std_train_score': array([0.00106707, 0.00119625, 0.00116904, 0.00141693, 0.00091399,\n",
       "        0.00120796])}"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "gsearch4_1.cv_results_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# summarize results\n",
    "print(\"Best: %f using %s\" % (gsearch4_1.best_score_,gsearch4_1.best_params_))\n",
    "\n",
    "test_means = gsearch4_1.cv_results_['mean_test_score']\n",
    "test_stds = gsearch4_1.cv_results_['std_test_score']\n",
    "train_means = gsearch4_1.cv_results_['mean_train_score']\n",
    "train_stds = gsearch4_1.cv_results_['std_train_score']\n",
    "\n",
    "pd.DataFrame(gsearch4_1.cv_results_).to_csv('my_preds_reg_alpha_reg_lambda_4.csv')\n",
    "\n",
    "# plot results\n",
    "test_scores = np.array(test_means).reshape(len(reg_alpha),len(reg_lambda))\n",
    "train_scores = np.array(train_means).reshape(len(reg_alpha),len(reg_lambda))\n",
    "\n",
    "for i, value in enumerate(reg_alpha):\n",
    "    pyplot.plot(reg_lambda,-test_scores[i],label='test_reg_alpha:'+str(value))\n",
    "    \n",
    "# for i, value in enumerate(reg_lambda):\n",
    "#    pyplot.plot(reg_alpha,train_scores[i],label='train_min_reg_lambda:'+str(value))\n",
    "\n",
    "pyplot.legend()\n",
    "pyplot.xlabel('reg_lambda')\n",
    "pyplot.ylabel('Log Loss')\n",
    "pyplot.savefig('reg_alpha_vs_reg_lambda_1.png')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "两个参数reg_alpha、reg_lambda，横轴只能一个参数（reg_lambda），另外参数（reg_alpha）用不同曲线表示。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Best: -0.588640 using {'reg_alpha': 1.5, 'reg_lambda': 1}"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.14"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
